Efficient allocation of network resources, such as available network bandwidth, has become critical as enterprises increase reliance on distributed computing environments and wide area computer networks to accomplish critical tasks. The widely-used Transport Control Protocol (TCP)/Internet Protocol (IP) protocol suite, which implements the world-wide data communications network environment called the Internet and is employed in many local area networks, omits any explicit supervisory function over the rate of data transport over the various devices that comprise the network. While there are certain perceived advantages, this characteristic has the consequence of juxtaposing very high-speed packets and very low-speed packets in potential conflict and produces certain inefficiencies. Certain loading conditions degrade performance of networked applications and can even cause instabilities which could lead to overloads that could stop data transfer temporarily.
In order to understand the context of certain embodiments of the invention, the following provides an explanation of certain technical aspects of a packet based telecommunications network environment. Internet/Intranet technology is based largely on the TCP/IP protocol suite. At the network level, IP provides a “datagram” delivery service—that is, IP is a protocol allowing for delivery of a datagram or packet between two hosts. By contrast, TCP provides a transport level service on top of the datagram service allowing for guaranteed delivery of a byte stream between two IP hosts. In other words, TCP is responsible for ensuring at the transmitting host that message data is divided into packets to be sent, and for reassembling, at the receiving host, the packets back into the complete message.
TCP has “flow control” mechanisms operative at the end stations only to limit the rate at which a TCP endpoint will emit data, but it does not employ explicit data rate control. The basic flow control mechanism is a “sliding window”, a window which by its sliding operation essentially limits the amount of unacknowledged transmit data that a transmitter is allowed to emit. Another flow control mechanism is a congestion window, which is a refinement of the sliding window scheme involving a conservative expansion to make use of the full, allowable window.
The sliding window flow control mechanism works in conjunction with the Retransmit Timeout Mechanism (RTO), which is a timeout to prompt a retransmission of unacknowledged data. The timeout length is based on a running average of the Round Trip Time (RTT) for acknowledgment receipt, i.e. if an acknowledgment is not received within (typically) the smoothed RTT+4*mean deviation, then packet loss is inferred and the data pending acknowledgment is re-transmitted. Data rate flow control mechanisms which are operative end-to-end without explicit data rate control draw a strong inference of congestion from packet loss (inferred, typically, by RTO). TCP end systems, for example, will “back-off,”—i.e., inhibit transmission in increasing multiples of the base RTT average as a reaction to consecutive packet loss.
A crude form of bandwidth management in TCP/IP networks (that is, policies operable to allocate available bandwidth from a single logical link to network flows) is accomplished by a combination of TCP end systems and routers which queue packets and discard packets when some congestion threshold is exceeded. The discarded and therefore unacknowledged packet serves as a feedback mechanism to the TCP transmitter. Routers support various queuing options to provide for some level of bandwidth management. These options generally provide a rough ability to partition and prioritize separate classes of traffic. However, configuring these queuing options with any precision or without side effects is in fact very difficult, and in some cases, not possible. Seemingly simple things, such as the length of the queue, have a profound effect on traffic characteristics. Discarding packets as a feedback mechanism to TCP end systems may cause large, uneven delays perceptible to interactive users. Moreover, while routers can slow down inbound network traffic by dropping packets as a feedback mechanism to a TCP transmitter, this method often results in retransmission of data packets, wasting network traffic and, especially, inbound capacity of a Wide Area Network (WAN) link. In addition, routers can only explicitly control outbound traffic and cannot prevent inbound traffic from over-utilizing a WAN link. A 5% load or less on outbound traffic can correspond to a 100% load on inbound traffic, due to the typical imbalance between an outbound stream of acknowledgments and an inbound stream of data.
In response, certain data flow rate control mechanisms have been developed to provide a means to control and optimize efficiency of data transfer as well as allocate available bandwidth among a variety of business enterprise functionalities. For example, U.S. Pat. No. 6,038,216 discloses a method for explicit data rate control in a packet-based network environment without data rate supervision. Data rate control directly moderates the rate of data transmission from a sending host, resulting in just-in-time data transmission to control inbound traffic and reduce the inefficiencies associated with dropped packets. Bandwidth management devices allow for explicit data rate control for flows associated with a particular traffic classification. For example, U.S. Pat. No. 6,412,000, above, discloses automatic classification of network traffic for use in connection with bandwidth allocation mechanisms. U.S. Pat. No. 6,046,980 discloses systems and methods allowing for application layer control of bandwidth utilization in packet-based computer networks. For example, bandwidth management devices allow network administrators to specify policies operative to control and/or prioritize the bandwidth allocated to individual data lows according to traffic classifications. In addition, certain bandwidth management devices, as well as certain routers, allow network administrators to specify aggregate bandwidth utilization controls to divide available bandwidth into partitions. With some network devices, these partitions can be configured to ensure a minimum bandwidth and/or cap bandwidth as to a particular class of traffic. An administrator specifies a traffic class (such as File Transfer Protocol (FTP) data, or data flows involving a specific user) and the size of the reserved virtual link—i.e., minimum guaranteed bandwidth and/or maximum bandwidth. Such partitions can be applied on a per-application basis (protecting and/or capping bandwidth for all traffic associated with an application) or a per-user basis (controlling, prioritizing, protecting and/or capping bandwidth for a particular user). In addition, certain bandwidth management devices allow administrators to define a partition hierarchy by configuring one or more partitions dividing the access link and further dividing the parent partitions into one or more child partitions. While the systems and methods discussed above that allow for traffic classification and application of bandwidth utilization controls on a per-traffic-classification basis operate effectively for their intended purposes, they possess certain limitations. As discussed more fully below, identification of traffic types associated with data flows traversing an access link involves the application of matching criteria or rules to explicitly presented or readily discoverable attributes of individual packets against an application signature which may comprise a protocol identifier (e.g., TCP, HyperText Transport Protocol (HTTP), User Datagram Protocol (UDP), Multipurpose Internet Mail Extensions (MIME) types, etc.), a port number, and even an application-specific string of text in the payload of a packet. After identification of a traffic type corresponding to a data flow, a bandwidth management device associates and subsequently applies bandwidth utilization controls (e.g., a policy or partition) to the data flow corresponding to the identified traffic classification or type. Accordingly, simple changes to an application, such as a string of text appearing in the payload or the use of encryption text may allow the application to evade proper classification and corresponding bandwidth utilization controls or admission policies.
Indeed, a common use of bandwidth management devices is to limit the bandwidth being consumed by unruly, bandwidth-intensive applications, such as peer-to-peer applications (e.g., Kazaa, Napster, etc.), and/or other unauthorized applications. Indeed, the rich Layer 7 classification functionality of Packetshaper® bandwidth management devices offered by Packeteer®, Inc. of Cupertino, Calif. is an attractive feature for network administrator, as it allows for accurate identification of a variety of application types. This traffic classification functionality, in many instances, uses a combination of known protocol types, port numbers and application-specific attributes to differentiate between various application traffic traversing the network. An increasing number of such peer-to-peer applications, however, employ data compression, encryption technology, and/or proprietary protocols that obscure or prevent identification of various application-specific attributes, often leaving well-known port numbers as the only basis for classification. In fact, as networked applications get increasingly complicated, data encryption has become a touted feature. Indeed, encryption addresses the concern of security and privacy issues, but it also makes it much more difficult to identify unauthorized applications using encryption, such as the peer-to-peer applications “Earthstation 5” and “Winny.” In addition, traffic classification based solely on well-known port numbers can be problematic, especially where the application uses dynamic port number assignments or an application incorrectly uses a well-known port number, leading to misclassification of the data flows. In addition, classifying such encrypted network traffic as “unknown” and applying a particular rate or admission policy to unknown traffic classes undermines the granular control otherwise provided by bandwidth management devices and, further, may cause legitimate, encrypted traffic to suffer as a result.
In addition, network savvy users (such as students in a campus or university environment) have also become aware that bandwidth management devices have been deployed to limit or restrict unauthorized peer-to-peer application traffic. As a result, users often attempt to bypass or thwart the bandwidth management scheme effected by such bandwidth management devices by creating communications tunnels (proxy tunnels) through which unauthorized or restricted network traffic is sent. The attributes discernible from the content of these tunneled data flows, however, often reveal little information about its true nature. For example, commercial HTTP tunnel services (such as loopholesoftware.com, TotalRc.net, and http-tunnel.com, etc.) allow users to send all network traffic in the form of HTTP traffic through a HTTP tunnel between a tunnel client and an HTTP proxy server maintained by the tunnel services provider. FIG. 6 illustrates the functionality and operation of a typical HTTP proxy tunnel. Client device 42 includes a client application (such as a peer-to-peer application 71) and a tunnel client 72. The client application sends data to the tunnel client 72 which tunnels the data over HTTP to a tunnel proxy server 74. The tunnel proxy server 74 then forwards the data to the intended destination (here, network resource 75), and vice versa. Such HTTP tunnels typically feature encryption; accordingly, a bandwidth management device 30, encountering the tunneled traffic in this form, may not detect the exact nature of the traffic and, in fact, classify such data flows as legitimate or regular HTTP traffic. Accordingly, these tunneling mechanisms and other techniques for evading bandwidth utilization controls implemented by bandwidth management devices present new challenges to network administrators and bandwidth management device manufacturers desiring to effectively control unauthorized or restricted network traffic.
In light of the foregoing, a need in the art exists for methods, apparatuses and systems that facilitate the classification of encrypted or compressed network traffic. A need further exists for methods, apparatuses and systems that facilitate the classification of network traffic associated with a non-public, proprietary protocol or application. Embodiments of the present invention substantially fulfill these needs.